\section{Experimental Results} \label{sec:experiments}
\noindent
We have conducted experiments to evaluate our proposed design
framework. We used a set of benchmarks constructed with inverted
pendulums, ball and beam processes, DC servos, and harmonic
oscillators~\cite{astrom97}. Such plants are acknowledged as
representatives of realistic control problems and are used extensively
for experimental evaluation by researchers in the real-time and
control systems communities. Our benchmarks varied in size between 5
and~9 computation nodes with 4 to 6 control applications. These may
represent clusters of nodes in automotive or avionics networks in
which each cluster comprises a set of nodes that run a certain class
of applications~\cite{navetbook09} (each cluster is treated separately
in our framework). All experiments were performed on a PC with a
quad-core CPU at 2.2~GHz, 8~GB of RAM, and running Linux.


As a baseline of comparison, we considered a straightforward design approach 
for which we synthesize solutions for all base configurations and the initial configuration
$\nodeset$. This constitutes the mandatory set of solutions to achieve fault tolerance in any feasible
configuration, as well as an optimized solution for the case when all nodes are operational.
We computed a cost $\controlcost^\textrm{base}$
according to Equation~\ref{eq:dseCost}, considering
that solutions have been synthesized for base configurations and the initial configuration, and
that all other feasible configurations run with the corresponding base configuration with the minimum
level of control quality given by Equation~\ref{eq:inheritedcost}.
The cost $\controlcost^\textrm{base}$ indicates the overall control quality of the fault-tolerant control system
with only the mandatory solutions synthesized.

Subsequently, we conducted experiments
with our optimization heuristic to select additional configurations
for synthesis. For each feasible configuration that is synthesized, individual cost terms in 
Equation~\ref{eq:dseCost} are decreased (control quality is improved compared to what is provided
by base configurations).
The optimization phase was conducted for varying amounts of design time. For each additional 
configuration that was synthesized, the total cost in Equation~\ref{eq:dseCost} was updated.
Reminding that a small control cost
indicates high control quality, and vice versa, we are interested in
the control-cost improvement $( \controlcost^\textrm{base} -
\controlcost ) / \controlcost^\textrm{base}$ relative to the control
cost $\controlcost^\textrm{base}$ obtained when only considering the
mandatory configurations. 

\begin{figure}
  \centering
  \includegraphics[width=0.45\textwidth]{plot/results}
  \caption{Relative cost improvements and runtimes of our proposed design approach.}
  \label{fig:results}
\end{figure}

Figure~\ref{fig:results} shows the design time on the horizontal axis
and the corresponding relative improvement on the vertical axis. 
The design time corresponding to the case of zero improvement
refers to the mandatory design phase of identification and synthesis
of base configurations. The mandatory design phase for base
configurations is around only 10~minutes, which is sufficient to cover
all fault scenarios and provide a minimum level of control
quality in any feasible configuration.
Any additional design time that is invested leads to improved control quality compared
to the already synthesized fault-tolerant solution.
For example, we can achieve an improvement of
30~percent already after 20~minutes. 
We did not run the heuristic for
the case of 5~nodes for more than 23 minutes, because at that time it
has already synthesized all feasible configurations. For the other
cases, the problem size was too large to afford an exhaustive
exploration of all configurations. It should be noted that the quality
improvement is smaller at large design times. At large design times,
the heuristic typically evaluates and optimizes control quality for
configurations with many failed nodes. However, these quality
improvements do not contribute significantly to the overall quality
(Equation~\ref{eq:dseCost}) beacuse the probability of many nodes
failing is very small (Equation~\ref{eq:failureprobability}).
Reminding that our optimization heuristic belongs to the class of anytime algorithms, meaning
that the algorithm can be terminated at any time and return a feasible solution,
we conclude that the designer can stop the design-space exploration process
when the improvement at each step is not significant.


